Virtualized Screen: A Third Element for Cloud-Mobile Convergence

In recent years, we have witnessed the strong growth of cloud computing, which refers to both applications delivered as services over the Internet and the infrastructure (including both hardware and software systems) in the data centers that supports those services. In general, cloud computing provides a centralized platform where programs can be executed and data can be stored. Cloud services built on the platform can offload many computing tasks normally solely done on the client devices and can be accessed instantly from anywhere at any time. However, until recently, the place where screen images that interface with users are rendered has been mostly done locally on client devices. Our argument is that the screen rendering can also be moved to the cloud and the rendered screen can be delivered as part of the cloud services. In general, the screen represents the whole or part of the display images. In a broad sense, it also represents a collection of data involved in user interfaces such as display images, audio data, mouse, keyboard, pen and touch inputs, and other multimodality inputs and outputs. In this article, we use the term screen to refer to the display images. There are a number of advantages to putting screen rendering in the cloud. First, because screen rendering is closely coupled with program execution and data storage, putting screen rendering in the cloud where data storage and program execution are centralized actually simplifies the cloud computing architecture. Second, screen rendering, especially graphics-rich screen rendering, is not a trivial task. It often requires a powerful CPU and GPU on the client devices. Moving screen rendering to the cloud will greatly reduce the hardware requirement for client devices and thus make low-cost devices possible. Third, even if the client devices are equipped with powerful CPU and GPU processors, offloading all or part of screen-rending task to the cloud will spare the client processing power to handle more effectively higher priority tasks (such as the local UI) and rich interactions (such as touch and gesture recognition) that require fast responses locally. Last, screen rendering in the cloud also brings a new avenue for optimizing the overall computing experience. Screen virtualization or screen rendering in the cloud doesn’t always mean putting the entire screen-rendering task in the cloud. Depending on the actual situations—such as local processing power, bandwidth and delay of the network, data dependency and data traffic, and display resolution—screen rendering can be partially done in the cloud and partially done at the clients (that is, scalable screen virtualization); collaboratively, a complete rendered screen is presented to the user. This is very similar to traditional cloud computing in which we have to decide where we should put the program execution and data storage, either remotely in the cloud or locally on the device, to obtain an optimized computing experience. [3B2-9] mmu2011020004.3d 11/4/011 11:42 Page 4